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Classroom response systems (CRSs) can be potent tools for teaching physics. Their efficacy, 
however, depends strongly on the quality of the questions used. Creating effective questions is 
difficult, and differs from creating exam and homework problems. Every CRS question should have 
an explicit pedagogic purpose consisting of a content goal, a process goal, and a metacognitive 
goal. Questions can be engineered to fulfil their purpose through four complementary mechanisms: 
directing students' attention, stimulating specific cognitive processes, communicating information 
to instructor and students via CRS-tabulated answer counts, and facilitating the articulation and 
confrontation of ideas. We identify several tactics that help in the design of potent questions, 
and present four "makeovers" showing how these tactics can be used to convert traditional physics 
questions into more powerful CRS questions. 
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I. INTRODUCTION 

Electronic classroom response systems (CRS) such as 
elnstruction CPS, InterWrite (formerly EduCue) PRS, 
and H-ITT can be powerful tools in the service of physics 
instruction. [1-6] They are merely tools, however, not a 
"magic bullet." To significantly impact student learn- 
ing, a CRS must be employed with skill in the service of 
sound, coherent pedagogy. This is not easy. 

In a research project entitled Assessing-to-Learn, we 
helped high school teachers learn to teach physics with a 
CRS and studied their progress and difficulties. [7-9] We 
have also used a CRS in our own university physics teach- 
ing, helped others learn to do so, and designed and tested 
CRS questions for more than ten years. [1] Our experience 
spans a broad array of contexts: high school classes, in- 
troductory univerisity classes for non-science and science 
majors, upper-level university classes for physics majors, 
and workshops for in-service physics teachers and for sci- 
ence graduate students. We've used a CRS with classes 
ranging in size from fewer than twenty to over two hun- 
dred students. We've taught traditional physics material, 
and also "conceptual" physics for non-scientists, general 
science, and science pedagogy. 

We can state unequivocally that learning to operate 
the technology is the easiest part of becoming facile with 
CRS-based instruction. More difficult challenges include 
creating and adapting suitable questions, cultivating pro- 
ductive classroom discourse, and integrating CRS use 
with the rest of the course, with curricular materials, 
and with external constraints. [9] 

Many who try teaching with a CRS discover that cre- 
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ating or finding "good" questions is more difficult than 
it at first appears. The characteristics of effective CRS 
questions are quite different from those of good exam 
questions, homework problems, and in-class worked ex- 
amples. The vast archives of questions and problems that 
instructors accumulate over years of teaching, or find in 
standard textbooks, therefore offer little assistance to the 
new CRS user. Few collections specifically designed for 
CRS-based teaching exist. 

One relatively widely-known collection of CRS ques- 
tions is that contained in Eric Mazur's book Peer 
Instruction. [3] Mazur has popularized CRS use in physics 
instruction, and the questions in his book make a useful 
starting point. However, we have found this collection 
to be insufficient for two reasons. One reason is that the 
questions have been designed to support Mazur's par- 
ticular goals for CRS use, and are not optimal for the 
more ambitious approach we use and advocate. Mazur's 
Peer Instruction method consists of brief lecture-style 
presentations on key points of physics, each followed by a 
short conceptual question. Students are asked to formu- 
late their own answers and then convince their peers of 
the correctness of their answer. Mazur argues that this 
process "forces students to think though the arguments 
being developed, and. . . provides them (as well as the 
teacher) with a way to assess their understanding of the 
concept" (Ref. 3, p. 10). He also suggests that knowl- 
edge "diffuses" among the students, helping to spread 
correct ideas. 

While this way of using a CRS is viable and valu- 
able, we find that even more dramatic improvements 
in teaching and learning can occur by inverting the 
paradigm. Rather than following mini-lectures with 
CRS-based quizzing, we use a CRS-powered "question 
cycle" as the core of in-class instruction, making question 
posing, pondering, answering, and discussing the vehicle 
of learning. Micro-lectures are injected only occasionally 
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and when immediate circumstances warrant. Further- 
more, we use a CRS to develop more than just concep- 
tual understanding. We also target the development of 
cognitive skills, analysis and problem solving ability, and 
productive student metacognition about physics, learn- 
ing, and thinking. Our approach, called Question-Driven 
Instruction, is summarized below in Subsection II A and 
described in more detail elsewhere. [1, 8, 10, 11] 

The other reason that the questions in Peer Instruction 
and similar collections are insufficient (although useful) is 
that instructors cannot use a question effectively if they 
do not appreciate its goals and design logic. They often 
fail to take advantage of the question's latent potential 
or unwittingly sabotage its effect. Instructors who would 
use a CRS well need a generalized understanding of what 
makes CRS questions succeed or fail, how to evaluate 
questions, and how to invent or adapt questions to meet 
their personal situation, objectives, and style. To put 
it bluntly, instructors should know how to create and 
modify, not just use, CRS questions. Although a few 
isolated question-design techniques have been developed 
and publicized, [12] we are aware of no comprehensive or 
systematic framework for developing and evaluating CRS 
questions. 

This paper addresses that need. Section II summa- 
rizes our vision of the central role that CRS use can play 
within physics instruction, and lays out a framework for 
thinking about and designing CRS questions. Section III 
describes a selection of specific tactics that can be em- 
ployed when designing questions. Section IV presents 
four "makeovers" : examples of traditional physics ques- 
tions, together with improved variations that implement 
some of the tactics of Section III. Section V summarizes 
the paper and offers some closing comments. 



II. THEORY: GOALS AND MECHANISMS 

A general framework for thinking about question de- 
sign must address the role questions will play within a 
course, the specific goals a question can be designed to 
attain, and the various mechanisms by which it can at- 
tain them. 



A. Role: What Part do Questions Play? 

We advocate a model of CRS-based teaching that we 
call Question- Driven Instruction (QDI). In this model, 
posing questions via CRS does more than augment tra- 
ditional instruction: it forms the very core of the instruc- 
tional dynamic. Our primary in-class goal is not to "lec- 
ture" or present information. Rather, we seek to help 
students explore, organize, integrate, and extend their 
knowledge. Students receive their primary exposure to 
new material from textbooks, multimedia, and other out- 
of-class resources. 
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FIG. 1: The question cycle used for Question-Driven Instruc- 
tion with a classroom response system. 



In-class activity is organized around a question cycle 
(Fig. 1).[8] We begin the cycle by presenting a question 
or problem to the class, generally without preamble, and 
allow a few minutes for students to discuss it in small 
groups. Typically, students within a group will argue 
their various opinions and intuitions, work out a solution 
if required, and continue discussing and elaborating until 
satisfied with their answer. Students then key in their re- 
sponses. We view and display an instant histogram show- 
ing the class-wide distribution of responses. Without re- 
vealing which responses are superior, we then moderate 
a class-wide discussion, asking for volunteers to explain 
the reasoning behind each. With deft management, this 
process can be turned into a lively interchange of ideas 
and arguments between students. Based on the thinking 
students reveal during discussion, we can follow up with 
general observations, a brief micro-lecture, a related CRS 
question, or whatever else is necessary for closure; at this 
point, a few well-chosen comments can often precipitate 
significant learning. We typically repeat the entire cycle 
three or four times per 50-minute class. 

Three aspects of the cycle are worth stressing here. 
First, questions are presented to students in a way that 
encourages significant cogitation, rather than just mem- 
ory recall or execution of practiced skills. Second, ques- 
tions are accompanied by extensive discussion: within 
small groups before answers are collected, and by the 
whole class afterward. Third, the instructor continually 
probes for and adjusts to the students' learning needs — a 
practice we call "agile teaching." 
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TABLE I: "Habits of mind" that expert physicists possess 
and students should develop. 



Basic 



Advanced 



Seek alternative representations 
Compare & contrast 
Explain, describe & depict 
Predict & observe 
Extend the context 



Generate multiple solutions 
Categorize & classify 
Discuss, summarize & model 
Strategize, justify & plan 



qualitative analysis in pursuit of an answer. 

Metacognitive goal: A question's metacognitive goal 
is chosen by answering the question, "What beliefs about 
learning and doing physics do we wish to reinforce?" Ev- 
erything that occurs within a course expresses, explicitly 
or implicitly, a perspective on physics, thinking, learn- 
ing, teaching, and how the educational "game" should 
be played. To cite just a few of the many issues this in- 
Is physics about memorizing and applying rules 



Reflect & evaluate , . 

eludes' 

Monitor & refine communication Think about thinking & learning 
^^^^^^^^^^^^^^^^=^^^^^^^^^^^^^^^^= and equations, or about reasoning and making sense of 

the physical world? Should students study and work in 
isolation, or is learning a cooperative and social process? 
Is attention to detail important, or is getting the general 
idea sufficient? Should conscientious and able students 
expect to understand material the first time it is pre- 
sented, or is confusion, resolution, and multi-pass lcarn- 



B. Goals: What Should the Question Accomplish? 

We strongly believe that every CRS question used in 
class should serve an explicit pedagogic objective. By 
"pedagogic objective," we mean more than just a par- 
ticular piece of physics content. For maximum benefit, 
we maintain that every question should have a threefold 
purpose consisting of a content goal, a process goal, and 
a metacognitive goal. 

Content goal: A question's content goal is deter- 
mined by answering the question, "What piece(s) of sub- 
ject material do we want to illuminate?" This dimension 
of a question's purpose is the most obvious, and needs lit- 
tle discussion other than to suggest that concepts, princi- 
ples, and their interrelationships make the best fodder for 
productive questions. Since QDI cannot explore all as- 
pects of a subject in the classroom time allotted a typical 
course, we focus question cycle iterations on the foun- 
dational ideas, core principles, crucial distinctions, and 
conceptual organization of the material at hand. With 
a robust understanding of these, students are well pre- 
pared to learn advanced topics and special cases through 
reading and homework assignments. 

Process goal: A question's process goal is chosen by 
answering the question, "What cognitive skills do we 
want students to exercise?" If the content goal refers 
to what physics material students must use to answer the 
question, the process goal refers to how they must use it. 
One might also call a process goal a "cognitive goal." 

In addition to knowledge about physics, expert physi- 
cists possess a wide range of skills that make their knowl- 
edge useful in various situations. We have identified 
twelve habits of mind that successful physicists practice 
and that students should be encouraged to develop. [8] 
For convenience, we separate them into "basic" and "ad- 
vanced" sets, summarized in Table I. A question's pro- 
cess goal can be characterized by the habits of mind it 
exercises. 

A crucial activity for students to engage in, spanning 
and integrating many habits of mind, is analysis: under- 
standing a situation by identifying the essential concepts 
and their relationships, and reasoning with these to draw 
conclusions. [13] The practice of analysis develops robust 
conceptual understanding and connects it to successful 
problem-solving ability. As part of their process goal, 
questions should frequently require students to perform 



ing necessary? How much self-directed activity should 
learning physics require? The answers to these questions 
may be obvious to most instructors, but they are not to 
students; consistency of message is crucial. 

The more constructive students' metacognitive per- 
spective is, the more efficiently they can learn what we 
are trying to teach. By influencing their perspective, 
we can significantly enhance learning in our courses, and 
help to prepare students for future learning throughout 
and beyond school. And recent thinking on the transfer 
of knowledge suggests that "preparation for future learn- 
ing" is the most durable learning outcome our instruction 
is likely to achieve. [14, 15] 



C. Mechanisms: How Can a Question Accomplish 
its Goals? 

A QDI question can fulfil its pedagogic purpose by 
way of four different general mechanisms or "channels" : 
through focusing students' attention by posing the ques- 
tion, through stimulating cognitive processes as students 
ponder the question, through feedback provided to stu- 
dents and instructor by collective viewing of the response 
histogram, and through articulation and confrontation of 
ideas during discussion. 

Questions can have a very powerful effect on students 
merely by being posed and pondered. The first two mech- 
anisms can be thought of as "What are they thinking 
about?" and "How are they thinking about it?" A ques- 
tion can direct students' attention to specific facts, fea- 
tures, ideas, conflicts, or relationships, bringing issues to 
conscious awareness. Sometimes merely looking at an 
idea, or looking at it from the right angle, is enough to 
spark understanding. Other times, it is a necessary but 
insufficient first step. Also, a question can stimulate stu- 
dents to exercise specific cognitive processes: habits of 
mind and the general practice of analysis. No question 
can force students to engage in any particular cognitive 
process, of course; mental engagement is always volun- 
tary. However, the design of a question can necessitate 
certain processes to reach a successful answer, and can 
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make the need for certain processes relatively obvious. 

The third and most obvious mechanism by which a 
CRS question can serve pedagogic ends is by communi- 
cating information about student responses. By seeing 
the histogram of answers entered, the instructor learns 
about students' understanding, and students learn about 
their classmates' thinking. If this information is not 
merely noted, but actually used by the instructor or 
students to make subsequent teaching and learning deci- 
sions, then response system use constitutes formative as- 
sessment: assessment to enhance, rather than evaluate, 
learning. And formative assessment is perhaps the most 
effective instructional "innovation" ever studied. [16-20] 

Inter-student and student-instructor discussion is the 
fourth mechanism by which questions can fulfil their de- 
sign objectives. The discussion that accompanies use of 
a question — within small groups before answering and 
class-wide after the histogram is displayed — is crucial to 
effective QDI. One reason is that the act of articulat- 
ing beliefs, perceptions, assumptions, expectations, un- 
derstanding, and reasoning is inherently valuable to stu- 
dents. Thinking is often ill-formed, nebulous, and incon- 
sistent. When a student must cast such thinking into 
language, especially the precise language of physics, such 
deficiencies become evident and must be redressed. 

Another reason is that discussion involves a confronta- 
tion of different perceptions, different analyses, and dif- 
ferent conclusions. Exposing students to their class- 
mates' thinking challenges their own and promotes learn- 
ing. Arguing and reconciling differences promotes yet 
more. Telling students what to think is notoriously inef- 
fective; eliciting their thinking, confronting it with alter- 
natives, and seeking resolution works better. 

Yet another reason is that whole-class discussion can 
reveal to the instructor far more about students' under- 
standing and difficulties than any single histogram, no 
matter how informative the question's distracters. We 
use these discussions to actively inform ourselves of the 
nature and causes of our students' errors. This also is a 
kind of formative assessment. 



III. TACTICS: IMPLEMENTING THE THEORY 

The previous section presented a general framework 
for thinking about the design of questions for Question- 
Driven Instruction using a classroom response system. 
In this section, we present some specific tactics (listed in 
Table II) that can be used to implement the framework. 
These are merely a helpful, representative set; others cer- 
tainly exist. They have been grouped according to which 
of the four mechanisms of effect they employ. Our con- 
vention is to present tactic names in italics and habits 
of mind "in quotes." (Some habits of mind are also tac- 
tic names; the formatting indicates which is meant in a 
particular context.) For many of the tactics, we have 
indicated which of the makeovers in Section IV employ 
them. 



TABLE II: Question design tactics. 

Tactics for directing attention and raising awareness: 

Remove inessentials 

Compare & contrast 

Extend the context 

Re-use familiar question situations 

Oops-go-back 

Tactics for stimulating cognitive processes: 

interpret representations 

Compare & contrast 

Extend the context 

Identify a set or subset 

Rank variants 

Reveal a better way 

Strategize only 

Include extraneous information 

Omit necessary information 

Tactics for formative use of response data: 

Answer choices reveal likely difficulties 

Use "none of the above" 

Tactics for promoting articulation discussion: 

Qualitative questions 

Analysis & reasoning questions 

Multiple defensible answers 

Require unstated assumptions 

Trap unjustified assumptions 

Deliberate ambiguity 

Trolling for misconceptions 



A. Tactics for Directing Attention and Raising 
Awareness 

Removing inessentials from a question is a general, 
obvious, and often neglected tactic for focusing students' 
attention where we want it. By this, we mean removing 
anything inessential to the instructor's pedagogic pur- 
pose for the question (not necessarily to the students' 
efforts to answer it). For example, we avoid having po- 
tentially distracting features in the question's situation 
or potentially distracting steps in the thinking students 
must do to reach an answer. A question with a quan- 
titative point, for example, is only weakened by requir- 
ing quantitative calculations that may distract students 
and divert their cognitive resources. Makeover C in Sec- 
tion IV demonstrates this tactic. 

Compare and contrast is another tactic for directing 
attention and awareness. By having students compare 
two things, their attention will naturally be drawn to the 
differences between them. One way to implement this is 
to pose a question that has students comparing multiple 
situations — physical arrangements, processes, conditions, 
etc. — to categorize or order them. Another is to describe 
a situation and ask about the effect of changing some 
aspect of it. A third is to present a sequence of two or 
more CRS questions, in which the situations are iden- 
tical but the query varies, or in which the same query 
is made about slightly different situations. Makeover A 
demonstrates a question comparing two situations. 
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Extending the context of a known idea or skill is a 
habit of mind, and also a valuable question design tactic. 
By asking a familiar question about an unfamiliar situ- 
ation, students' attention is drawn to the ways in which 
the new situation differs from known ones and to the rel- 
evance of these differences, and broadens students' com- 
prehension of the relevant ideas. Students expand their 
understanding of concepts beyond the limited scope they 
are initially encountered in by seeing them in varied con- 
texts. For example, when students begin to grasp the 
procedure for finding the normal force of an object on a 
flat horizontal surface, tilt or curve the surface. When 
they assimilate that, add additional forces on the object 
or give it a nonzero acceleration. Makeover A incorpo- 
rates this tactic by asking a linear dynamics question 
about a situation generally used for rotational dynamics. 

Re-using familiar question scenarios also has its 
place. Reading, digesting, and interpreting a question 
statement requires nontrivial cognitive resources from 
students — resources that could be spent on understand- 
ing the points we want to make with the question. The 
tactic of removing inessentials helps reduce this "cogni- 
tive load" effect, as docs building new questions from sit- 
uations and systems students have already come to terms 
with. Use of the extending the context and re-using fa- 
miliar question scenarios tactics should be balanced: in 
general, new ideas should be introduced in familiar ques- 
tion contexts, while somewhat familiar ideas should be 
explored and developed in novel contexts. 

Oops-go-back is an awareness-raising tactic involving 
a sequence of two related questions. The first is a trap: 
a question designed to draw students into making a com- 
mon error or overlooking a significant consideration. The 
instructor allows students to respond, and then moves to 
the second question without much discussion. The sec- 
ond causes students to realize their mistake on the first. 
When students are "burned" this way by a mistake and 
discover it on their own, they are far more likely to learn 
from it than if they are merely warned about it in advance 
or informed that they have committed it. A simple exam- 
ple, suitable early during coverage of kinematics, would 
be to ask students for the velocity of some object moving 
in the negative direction or in two dimensions, with pos- 
itive scalar answer choices including the object's speed, 
and also "None of the above." Many, insufficiently at- 
tuned to the distinction between speed and velocity, will 
erroneously select the speed. Then, a second question 
asks about the object's speed, causing many students to 
consider how this question differs from the previous and 
realize their error. 

This can be a subtle technique, so we will present a 
second, less trivial example. If we ask the question in 
Fig. 2a, many students will erroneously answer "45 de- 
grees." Without discussion, we then present the question 
in Fig. 2b. Many will realize that the answer to this one 
is "45 degrees," be bothered by this, reconsider how the 
first question differed, and realize that they had neglected 
the cannon's velocity relative to the ground. While an- 
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FIG. 2: A question pair exemplifying the oops-go-back design 
tactic. 



swering the first question, they had unwittingly answered 
the second. 



B. Tactics for Stimulating Cognitive Processes 

Specific question design tactics for stimulating cogni- 
tive processes are as varied as the spectrum of thinking 
skills they target. The fundamental rule is to ask ques- 
tions which cannot be answered without exercising the 
desired habits of mind, and to avoid excess baggage that 
might distract students from the need to exercise them. 
Certain types of problems are helpful in this regard. 

Many students arc inordinately attached to algebraic 
representations of physics concepts, relationships, and 
situations and do not fully appreciate the utility of "al- 
ternative" representations such as graphs, free body and 
vector diagrams, and even verbal descriptions. Questions 
that require students to interpret representations are 
useful for remedying this and for developing the habit of 
mind "seek alternative representations." The tactic is 
implemented by providing necessary information or an- 
swer options in such an alternative representation. For 
example, one might describe an object's motion with a 
graph and then ask a question about its behavior that 
requires students to recognize and interpret the informa- 
tion latent in aspects of the graph, such as a slope or the 
area under a curve. Alternatively, one might ask them to 
verbally describe the meaning of a mathematical equa- 
tion, or to choose which of a set of vectors best describes 
some quantity. Makeover D does not require this of stu- 
dents, but it does rely on the tactic for the "surprise" 
solution revealed during discussion (cf. reveal a better 
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way, below). 

The compare and contrast and extend the con- 
text tactics for focusing students' attention, described 
above, are also useful for developing the habits of mind 
of those names. These are powerful question types with 
multiple benefits. 

Some habits of mind are easy to target. "Categorize 
and classify" is promoted by presenting students with a 
set of situations, objects, processes, or other things and 
asking them to identify a set or subset meeting some 
criterion, or to rank variants according to some quality. 

Constrain the solution is a tactic for exercising the 
habit of mind "generate multiple solutions." This can be 
a "positive constraint" directing students to solve a prob- 
lem via some particular approach (e.g., "use the work- 
energy theorem"), or a "negative constraint" directing 
students not to use some particular approach (e.g., "do 
it without solving for the acceleration"). Merely giving 
such a directive gets students to consider their activity 
from a strategic perspective. 

Reveal a better way is another, less direct tac- 
tic for strengthening "generate multiple solutions." One 
presents a question which students are likely to solve by a 
valid but difficult, tedious, error-prone, opaque, or other- 
wise non-optimal path. Then, during discussion, one can 
suggest a dramatically more elegant or simple solution. 
Makeover D is a classic example of this tactic. 

Strategize only is a tactic for strengthening the habit 
of mind "strategize, justify and plan." It is implemented 
by presenting a problem and asking students to iden- 
tify the principle (s) and approach that would be most 
useful for reaching a solution, without actually solving 
the problem. This teaches students to think explicitly 
about problem solving and the set of strategies available 
to them. (The "justify" portion of that habit of mind 
is naturally developed during discussion of students' rea- 
soning.) Makeover C demonstrates this. 

Include extraneous information and omit neces- 
sary information are other tactics useful for develop- 
ing "strategize, justify and plan." They push students to 
consider explicitly what information is necessary to com- 
plete a strategy, rather than assuming every question pro- 
vides exactly what is required and nothing more. (Note 
that include extraneous information is not inconsistent 
with remove inessentials, since deliberately extraneous 
information can be essential to the pedagogic purpose of 
the question. It is extraneous to students, but not to the 
instructor.) 

Similar tactics can be imagined for other habits of 
mind. Once one has decided to target a specific cogni- 
tive facility and has escaped from the trap of always using 
standard calculate-an-answer questions, creating suitable 
question types is generally straightforward. 

Cognitive processes are targeted not just by the in- 
trinsic construction of the question, but also by class- 
room interaction surrounding its use. Reveal a better 
way, described above, relies on this. "Monitor and re- 
fine communication" is exercised any time communica- 



tion within the classroom is explicitly discussed, perhaps 
after students have misinterpreted a question or a state- 
ment by the instructor. Similarly, "think about thinking 
and learning" (metacognition) is stimulated whenever the 
instructor asks students to explicitly consider their own 
thinking processes and learning behaviors. 



C. Tactics for Formative Use of Response Data 

As described above, the third general mechanism by 
which questions can fulfill their design objectives is by 
providing information to the instructor and students 
through the histogram of students' responses. To provide 
maximally useful information to the instructor, ques- 
tions should be designed so that answer choices reveal 
likely student difficulties: common errors, misunder- 
standings, and alternative assumptions and interpreta- 
tions. That way, by glancing at the histogram, we can 
quickly detect whether a particular one of these is preva- 
lent in our class and decide whether to address it. In 
general, a response histogram is most useful to students 
and instructor when the spectrum of answers chosen is 
broad rather than narrowly peaked around one choice. 
(One exception is the first question of an oops-go-back 
pair, for which having a majority of students fall into the 
"trap" can be desirable.) 

When interpreting students' responses, one must re- 
member that any given answer can almost always be 
reached by more than one path or argument. Thus, hav- 
ing students explain their answers is vital. For this reason 
among others, we usually open the whole-class discussion 
for a question by proceeding systematically down the an- 
swer list, asking for (or cajoling) volunteers to present a 
rationale for each. We maintain a "poker face" through- 
out. After a student has argued for a particular answer, 
we ask if anyone has a different reason for the same an- 
swer. 

We frequently include "none of the above" (or 

"not enough information") as an answer choice, so as 
to learn about responses we might not have anticipated. 
We make this the "correct" or best answer often enough 
that students learn to take it seriously (often for omit 
necessary information, described above) and don't as- 
sume that they have made a mistake if they don't agree 
with one of the other options offered. 



D. Tactics for Promoting Articulation, Conflict 
and Productive Discussion 

We said earlier that the fourth mechanism of ques- 
tion efficacy — discussion — has students learn by articu- 
lating their thinking, confronting each others' thinking, 
and resolving differences. It also provides the instructor 
with valuable information about students' understand- 
ing, confusions, progress, and predilections. Not all ques- 
tions lead to equally productive discussion. Questions 
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that are most useful for this tend to be quite different 
from standard exam-type questions. 

Qualitative questions are usually superior to quan- 
titative ones for promoting articulation and argument. 
Quantitative questions lure students into thinking in 
terms of numbers, variables, and equations, which are 
difficult to communicate and discuss; qualitative ques- 
tions promote discussion in terms of concepts, ideas, and 
general relationships. The final question versions in all 
four makeovers of Section IV are qualitative. 

Analysis and reasoning questions, requiring signif- 
icant decision-making by students, similarly lead to bet- 
ter discussion and more valuable articulation than those 
requiring calculation or memory recall. (They also pro- 
mote the development of analytic skills.) Makeovers B 
and C introduce elements of analysis, and D is a good 
analysis question (although it may initially seem like a 
straightforward algebra problem to students). 

Questions with multiple defensible answers arc 
useful for sowing dissension and generating productive 
discussion. Perhaps more than one answer is viable de- 
pending on how one chooses to interpret the question, or 
on what assumptions one consciously or unconsciously 
makes. Makeover B exemplifies this. 

Similarly, one can design questions that require un- 
stated assumptions, trap unjustified assumptions, 
or contain deliberate ambiguity. In addition to pro- 
moting disagreement and therefore profitable discussion, 
these have the benefit of sensitizing students to the mul- 
tiple interpretations possible for many situations, to the 
importance of assumptions in physics reasoning, and to 
the criteria physicists use when evaluating assumptions. 
Makeover D introduces an unstated assumption (leading 
to multiple defensible answers). 

Trolling for misconceptions is another useful tactic: 
engineering questions that deliberately catch students in 
likely misconceptions or undesirable "alternative concep- 
tions." Such questions further the content goal of help- 
ing students become aware of and escape the particular 
misconception, improving their physics knowledge. They 
further the metacognitive goal of putting students on the 
alert for misconceptions in general. They also tend to 
promote argument, sometimes impassioned. Makeovers 
A and B both target specific misconceptions. 

How one conducts class discussion can be more impor- 
tant to the quality of the discussion than what questions 
are used. Emphasizing cogency of reasoning over cor- 
rectness of results is crucial, as is stressing that the only 
"bad" answer is one that does not reflect a student's ac- 
tual thinking. Other, more specific tactics for moderating 
discussion exist, but are outside the scope of this paper. 



IV. EXAMPLES 

Abstract advice, divorced from concrete examples, can 
be difficult to implement. In this section we present four 
"makeovers" : case studies in which a traditional question 



is improved by incorporating some of the QDI question 
design tactics into it. Each is accompanied by a discus- 
sion of which tactics have been implemented. 



A. Newton's Second Law 

Fig. 3a shows a straightforward question on Newton's 
Second Law in one dimension. The variant in Fig. 3b 
requires the same content knowledge, but incorporates 
the tactic of trolling for a misconception to promote dis- 
agreement and argumentation. A common misconcep- 
tion among novice physics students is that r = la some- 
how supersedes or modifies F = ma. By displaying the 
surface features of a rotational dynamics problem, this 
question will lure many students into the trap of think- 
ing that because the disk rotates, some of the force is 
"used up" and the resulting linear acceleration will be 
less than Newton's Second Law alone would predict. Al- 
though requiring only Newton's Second Law in one linear 
dimension, the question would be appropriate for use af- 
ter rotational dynamics has been introduced. 

Note that the question wording does not explicitly ask 
for the linear acceleration of the disk. Most likely, some 
students will assume that the question is asking for angu- 
lar acceleration and choose the "cannot be determined" 
answer. This allows the instructor to stress that "accel- 
eration," unqualified, means "linear acceleration." If a 
large enough fraction of the class answers this way, we 
recommend clarifying the point, and then starting over 
without further discussion and allowing students to rc- 
answer so that the question's primary intent can be re- 
alized. (To make the variant more effective at trapping 
this error, the disk's radius can be given, and some an- 
swer choices in appropriate units for angular acceleration 
can be provided.) 

Fig. 3c shows a variant incorporating yet more ques- 
tion design tactics. Compare and contrast is used to fo- 
cus students' attention on the effect of rotational mo- 
tion on linear acceleration, and to practice the "compare 
and contrast" habit of mind. This variant is a qualita- 
tive question that trolls for the same misconception and 
interpretation error as the previous variant, but more ef- 
fectively. It is a powerful tool for pushing students to ar- 
ticulate an intuitive misconception and to realize, wrestle 
with, and resolve the contradictions it leads to. (With 
deft handling, an instructor can use it to compare and 
contrast the ideas of force and acceleration with work 
and energy: although the two disks experience the same 
force and have the same acceleration, one gains more ki- 
netic energy than the other in a given time interval.) 

The variant would have been effective, and simpler, if 
the three answer choices were "a^ < as," "oa = as," 
and u oa > as-" Both arc implementations of rank vari- 
ants. However, the set of distracters provided in variant 
b helps the instructor distinguish between students who 
think the disk will accelerate more slowly when rotating 
and those who think it will spin without translating at 
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FIG. 3: Three variants of a question on Newton's Second Law 
in one dimension. 



all, making use of answer choices reveal likely difficulties. 



B. Identifying Forces 

Fig. 4a shows a question targeting students' ability to 
identify the forces on a body. The question's content 
goal is to have students appreciate that apart from grav- 
ity (and other "action at a distance" forces not encoun- 
tered during introductory mechanics), all forces acting 
on a body are caused by interactions with other bodies 
in direct contact with it. This is a qualitative question 
that trolls for the misconception that contact forces can 
be "transmitted" through an intermediate body to act 



a) block with, a spring attached ond extended is pulled! 
up a rough surface by o string connected to a mass os 

ch of the following forces >s not exerted on it? 

weight of black m 
2) The weight of the hanging ' 



3) The spring force 
4} Kinetic friction with the 
incline 

5) The normal force of the incline 
6} Air resistance 
7} None of tht above 




fa) A block with a spring attached and extended is pulled 
up a rough surface by o struigi connected to o mass os 
shown. How nww forces ore exerted on It? 

I ) One force 

£) Two farces 

3J Three forces 

4) fan forces 

5) Five forces 
6} Si* farces 




3} Wore than seven forces 



FIG. 4: Two variants of a question on identifying forces. 



between two separated bodies. 

As written, the question is not bad, but it could be 
better. Consider the variant in Fig. 4b. It is still quali- 
tative and trolls for the same misconception. However, it 
is open-ended and has students enumerate the forces (a 
modification of identify a set or subset) to unearth other, 
perhaps unanticipated misconceptions and errors. For 
example, one might discover that some students count 
the "net force" alongside other forces. 

More importantly, it employs the multiple defensible 
answers tactic: choices 4 through 8 are all justifiable, de- 
pending on whether one treats the interaction between 
the plane and block as one contact force or as two (fric- 
tion and normal), whether one neglects buoyancy and 
drag due to air, whether one includes "silly" but real 
forces like the gravitational effect of the moon, etc. (In 
fact, from a microscopic perspective, one can argue for an 
uncountably large and fluctuating number of forces due 
to molecular collisions.) This could also be considered 
an implementation of the require unstated assumptions 
or deliberate ambiguity tactics. Knowing what answer a 
student has picked conveys little information about their 
degree of understanding or about specific confusions they 
may have. Instead, the question serves to get students 
thinking about which forces are present without prompt- 
ing them with specific forces. Then, during whole-class 
discussion, the instructor can ignore the answers chosen 
and instead discuss various possible forces in turn, argu- 
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ing whether or not each merits counting. 

This variant also addresses additional content goals. 
During discussion, the instructor can model a general 
procedure for identifying the forces on a body, and illumi- 
nate the various choices and conventions involved in iden- 
tifying a "force" (for example, the convention of treating 
one contact force between two surfaces as two distinct, 
orthogonal forces: the "normal" and "friction" forces). 
In addition, the question makes an excellent platform for 
discussing the role of assumptions and approximations 
in physics and helping students learn when to include or 
neglect various forces. 

In our minds, perhaps the most powerful aspect of this 
variant is its effectiveness at achieving the metacogni- 
tive goal of communicating to students that they should 
be concerned with reasoning, learning, and the cogency 
of their answers, and not with the correctness or incor- 
rectness of any particular answer choice. Most students 
are deeply attached to the notion that every question 
has a "right" or "best" answer. We have found that the 
only way to make students abandon this and really focus 
on reasoning is to use questions like 4b, for which it is 
patently obvious that several answers are defensible and 
can be "correct" or "incorrect" depending on the argu- 
ment behind them. 



C. Energy and Angular Motion 

Fig. 5a shows a question that helps students integrate 
their knowledge by requiring a mix of energy and angular 
motion ideas. To answer the question correctly, students 
must recognize the need for conservation of energy, apply 
it, and relate linear to angular motion. 

The variant in Fig. 5b improves on the original by in- 
corporating several question design tactics. Since the 
question's primary content goal is to develop students' 
ability to recognize the need for two different strategic 
steps using physics from two distinct topic areas (energy 
conservation and relating linear to angular motion), this 
variant is a qualitative question that removes inessentials 
to focus students' attention more effectively; students 
rarely pay sufficient attention to high-level, strategic as- 
pects of problem solving when embroiled in equation ma- 
nipulation. It uses the strategize only tactic. The ques- 
tion's phrase "most efficiently" can be considered a delib- 
erate ambiguity: is efficiency defined in terms of number 
of lines of calculation required, number of principles in- 
volved, intricacy of thought entailed, or something else? 
Does efficiency depend on the skills of the learner? (If the 
question initiates a class-wide discussion of "efficiency" 
in problem solving, so much the better.) 



D. Kinematics 

Fig. 6a is a relatively straightforward kinematics ques- 
tion. It is nontrivial, in that students aren't given the ac- 
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FIG. 5: Two variants of a question on strategic choices in 
problem solving. 



celcration and must determine it first before calculating 
the distance traveled. Alternatively, students can deter- 
mine the average velocity and multiply that by the time. 
The question requires some strategic decision-making. 

The variant in Fig. 6b is similar in content. If ap- 
proached algebraically, it is also similar in difficulty 
However, this variant is a qualitative question, making 
it more amenable to analysis and reasoning and more 
suitable for discussion. By omitting the statement "As- 
suming a constant acceleration," we have employed the 
require unstated assumptions tactic (and multiple defen- 
sible answers as a result) and opened up the possibility 
of discussing whether and how the constancy of the ac- 
celeration matters. 

More interestingly, this variant permits the instructor 
to reveal a better way of answering the question. Assume 
constant acceleration and sketch a graph of velocity vs. 
time. By identifying the area under the line (the time 
integral) with the distance traveled, one can use simple 
geometry to see that when the marble has slowed to half 
its original velocity, it has traveled three-quarters of the 
distance it will cover before coming to rest. If it reaches 
half speed after 20 cm/s, it must stop at 4/3 that dis- 
tance, which is before the end of the felt. This alter- 
nate solution involves interpreting representations, and 
the variant is a better analysis and reasoning question 
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V. SUMMARY & REMARKS 

In the Question Driven Instruction approach, using a 
classroom response system to pose, collect answers for, 
and discuss questions forms the core of in-class instruc- 
tion. The success of the approach depends in part on 
the quality of the questions used. Effective questions 
should be designed with an explicit, threefold pedagogic 
objective consisting of a content goal, a process (cogni- 
tive) goal, and a metacognitive goal. The content goal 
is the topic material to be addressed; it should generally 
be conceptual and foundational in nature, and should 
frequently integrate ideas from various portions of the 
curriculum. The process goal is the set of cognitive skills 
to be developed, and can be thought of in terms of twelve 
"habits of mind" and the general practice of qualitative 
analysis. The metacognitive goal is the perspective about 
learning or physics to be reinforced. 

A question can fulfil its threefold pedagogic objective 
through four complementary mechanisms. The posing of 
the question can focus students' attention on particular 
issues. Students' pondering of the question can stimu- 
late particular cognitive skills. Displaying the answer his- 
togram can convey information about student knowledge 



and thinking to classmates and to the instructor. And 
discussion, both small-group and whole-class, can impact 
students and inform the teacher as students struggle to 
articulate and defend their thinking and confront others' 
perceptions, interpretations, assumptions, and reasoning. 

Questions can be deliberately engineered for maximal 
learning, and we have identified some tactics such as re- 
move inessentials, compare and contrast, interpret repre- 
sentations, and strategize only that can be used in the 
design of powerful questions. Often a question in the 
"standard" style can be improved through minor mod- 
ifications that take advantage of one or more of these 
tactics. 

Even with an explicit framework such as the one pre- 
sented here, designing effective questions is challeng- 
ing and time-consuming, and — like any other nontriv- 
ial skill — requires practice. A repository of well-designed 
questions can be quite helpful, and we have made many of 
our questions available through both a public website [21] 
and an annotated commercial product. [22] However, to 
teach most effectively with another person's questions, 
one should understand the goals and design logic of each 
one. [9] The framework herein helps to analyze existing 
questions as well. 

We reiterate that well designed questions are merely a 
tool, one component of the QDI approach. Pedagogy — 
how the instructor uses questions to interact with stu- 
dents in the classroom — is more important. [8, 10, 11] 
Nevertheless, lack of effective questions can be a seri- 
ous and frustrating barrier to teachers seeking to learn 
and practice QDI. We believe the framework and ques- 
tion design tactics presented here can help overcome this 
barrier. 

Although a classroom response system is a tremen- 
dously useful tool for implementing QDI, it is not essen- 
tial to the philosophy underlying QDI. Many of the ideas 
presented here can be productively employed without 
technology, especially in small, highly interactive classes. 

QDI is a type of formative assessment. By is very 
nature, formative assessment tends to be self-correcting: 
the feedback it provides to the practitioner about student 
learning can, if studied attentively, reveal implementa- 
tion weaknesses and improve practice over time. There- 
fore, our most important piece of advice regarding QDI 
is to pay critical attention to what happens when you do 
it. Your students are your best teachers. 
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